Catch trial performance

## [1] "Excluded 1 participants based on catch-trial performance."

Exclusion of random guesses

We further exclude participants who seem to provide random ratings independent of the scene that they are seeing. We quantify this by computing the mean rating for each utterance across all trials for each participant and computing the correlation between a participant’s actual ratings and their mean rating. A high correlation is unexpected and indicates that a participant chose ratings at random. We therefore also exclude the data from participants for whom this correlation is larger than 0.75.

## `summarise()` has grouped output by 'modal'. You can override using the
## `.groups` argument.
## `summarise()` has grouped output by 'modal', 'percentage_blue'. You can
## override using the `.groups` argument.
## [1] "Excluded 0 participants based on random responses."

Aggregated results

## `summarise()` has grouped output by 'workerid', 'percentage_blue', 'modal'. You
## can override using the `.groups` argument.
## `summarise()` has grouped output by 'percentage_blue', 'modal'. You can
## override using the `.groups` argument.
## `summarise()` has grouped output by 'workerid', 'percentage_blue', 'modal'. You
## can override using the `.groups` argument.

Comparison across conditions

## `summarise()` has grouped output by 'workerid', 'percentage_blue', 'modal'. You
## can override using the `.groups` argument.
## `summarise()` has grouped output by 'percentage_blue', 'modal'. You can
## override using the `.groups` argument.

Individual responses

AUC computation

We use the AUC function with the splines method to directly compute the AUC.

t-test and regression model with control variables:

## 
##  Two Sample t-test
## 
## data:  aucs.cautious$auc_diff and aucs.confident$auc_diff
## t = 2.9517, df = 122, p-value = 0.003791
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##   3.969716 20.138118
## sample estimates:
## mean of x mean of y 
##  20.20750   8.15358
## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: 
## auc_diff ~ cond + test_order + first_speaker_type + confident_speaker +  
##     first_speaker_type * cond + (1 | workerid)
##    Data: auc_d
## 
## REML criterion at convergence: 1086.2
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -3.1828 -0.4581  0.0770  0.6374  2.0920 
## 
## Random effects:
##  Groups   Name        Variance Std.Dev.
##  workerid (Intercept)  89.25    9.447  
##  Residual             376.89   19.414  
## Number of obs: 124, groups:  workerid, 62
## 
## Fixed effects:
##                           Estimate Std. Error     df t value Pr(>|t|)    
## (Intercept)                 14.152      2.118 58.000   6.684 9.95e-09 ***
## cond1                        6.027      1.743 60.000   3.457  0.00101 ** 
## test_order1                  0.878      2.122 58.000   0.414  0.68056    
## first_speaker_type1         -7.666      2.118 58.000  -3.620  0.00062 ***
## confident_speaker1           2.544      2.122 58.000   1.199  0.23544    
## cond1:first_speaker_type1    1.211      1.743 60.000   0.695  0.48990    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) cond1  tst_r1 frs__1 cnfd_1
## cond1        0.000                            
## test_order1 -0.032  0.000                     
## frst_spkr_1  0.000  0.000  0.002              
## cnfdnt_spk1  0.002  0.000 -0.065 -0.032       
## cnd1:frs__1  0.000  0.000  0.000  0.000  0.000

Clustering analyses

library(mclust)
## Package 'mclust' version 5.4.10
## Type 'citation("mclust")' for citing this R package in publications.
## 
## Attaching package: 'mclust'
## The following object is masked from 'package:DescTools':
## 
##     BrierScore
## The following object is masked from 'package:bootstrap':
## 
##     diabetes
aucs_diff = merge(aucs.cautious, aucs.confident, by=c("workerid"))
aucs_diff$diff_of_diffs = aucs_diff$auc_diff.x - aucs_diff$auc_diff.y

aucs_diff %>% ggplot(aes(x=diff_of_diffs)) + geom_density() + geom_jitter(aes(y=0), width=0, height=0.001)  + ggtitle("Raw data + estimated density")

Gaussian mixture models of diffeences of AUC differences

1 Cluster

fit1 = Mclust(aucs_diff$diff_of_diffs, G=1)
print(summary(fit1, parameters=2))
## ---------------------------------------------------- 
## Gaussian finite mixture model fitted by EM algorithm 
## ---------------------------------------------------- 
## 
## Mclust X (univariate normal) model with 1 component: 
## 
##  log-likelihood  n df      BIC      ICL
##       -292.5844 62  2 -593.423 -593.423
## 
## Clustering table:
##  1 
## 62 
## 
## Mixing probabilities:
## 1 
## 1 
## 
## Means:
## [1] 12.05392
## 
## Variances:
## [1] 735.3371

2 Clusters

fit2 = Mclust(aucs_diff$diff_of_diffs, G=2)
print(summary(fit2, parameters=T))
## ---------------------------------------------------- 
## Gaussian finite mixture model fitted by EM algorithm 
## ---------------------------------------------------- 
## 
## Mclust E (univariate, equal variance) model with 2 components: 
## 
##  log-likelihood  n df       BIC       ICL
##       -288.1198 62  4 -592.7482 -598.9994
## 
## Clustering table:
##  1  2 
## 55  7 
## 
## Mixing probabilities:
##        1        2 
## 0.864934 0.135066 
## 
## Means:
##         1         2 
##  4.335758 61.479346 
## 
## Variances:
##        1        2 
## 353.8638 353.8638

3 Clusters

fit3 = Mclust(aucs_diff$diff_of_diffs, G=3)
print(summary(fit3, parameters=T))
## ---------------------------------------------------- 
## Gaussian finite mixture model fitted by EM algorithm 
## ---------------------------------------------------- 
## 
## Mclust E (univariate, equal variance) model with 3 components: 
## 
##  log-likelihood  n df       BIC       ICL
##       -288.1317 62  6 -601.0263 -656.7775
## 
## Clustering table:
##  1  2  3 
##  9 46  7 
## 
## Mixing probabilities:
##         1         2         3 
## 0.3373576 0.5314408 0.1312016 
## 
## Means:
##         1         2         3 
## -1.081510  7.998816 62.254385 
## 
## Variances:
##        1        2        3 
## 337.7512 337.7512 337.7512

According to the Bayesian information criterion, a model with two clusters describes the data best.

Fitted model:

aucs_diff %>% 
  ggplot(aes(x=diff_of_diffs)) + 
    geom_jitter(aes(y=0, color=first_speaker_type.x), width=0, height=0.001)  +
    ggtitle("Raw data + Components of gaussian mixture") + 
    stat_function(fun = dnorm, args = list(mean = fit2$parameters$mean[1], sd = sqrt(fit2$parameters$variance$sigmasq[1]))) + 
    stat_function(fun = dnorm, args = list(mean = fit2$parameters$mean[2], sd = sqrt(fit2$parameters$variance$sigmasq[2])))
## Warning: Removed 101 row(s) containing missing values (geom_path).

Compute likelihoods based on the adaptation model

## Generalized linear mixed model fit by maximum likelihood (Laplace
##   Approximation) [glmerMod]
##  Family: binomial  ( logit )
## Formula: most_likely_model ~ condition + test_order + first_speaker_type +  
##     first_speaker_type * condition + (1 | workerid)
##    Data: d.post_test
## 
##      AIC      BIC   logLik deviance df.resid 
##    154.1    171.0    -71.0    142.1      118 
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -1.6887 -0.5686 -0.2379  0.5566  2.2970 
## 
## Random effects:
##  Groups   Name        Variance Std.Dev.
##  workerid (Intercept) 1.849    1.36    
## Number of obs: 124, groups:  workerid, 62
## 
## Fixed effects:
##                                              Estimate Std. Error z value
## (Intercept)                                   -0.4481     0.3106  -1.443
## conditioncautious                             -0.8698     0.2906  -2.993
## test_orderparallel                            -0.4682     0.3081  -1.520
## first_speaker_typecautious                     0.8680     0.3414   2.542
## conditioncautious:first_speaker_typecautious  -0.2066     0.2423  -0.852
##                                              Pr(>|z|)   
## (Intercept)                                   0.14906   
## conditioncautious                             0.00276 **
## test_orderparallel                            0.12856   
## first_speaker_typecautious                    0.01102 * 
## conditioncautious:first_speaker_typecautious  0.39396   
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) cndtnc tst_rd frst__
## conditincts  0.209                     
## tst_rdrprll  0.106  0.220              
## frst_spkr_t -0.193 -0.363 -0.198       
## cndtncts:__ -0.039  0.080  0.055 -0.062
## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: likelihood_ratio ~ condition + test_order + first_speaker_type +  
##     first_speaker_type * condition + (1 | workerid)
##    Data: d.post_test
## 
## REML criterion at convergence: 1596.8
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -2.2024 -0.6789 -0.0483  0.5532  2.6459 
## 
## Random effects:
##  Groups   Name        Variance Std.Dev.
##  workerid (Intercept)  2007     44.8   
##  Residual             30231    173.9   
## Number of obs: 124, groups:  workerid, 62
## 
## Fixed effects:
##                                              Estimate Std. Error     df t value
## (Intercept)                                    -14.48      16.63  59.00  -0.871
## conditioncautious                              -56.74      15.61  60.00  -3.634
## test_orderparallel                             -25.21      16.63  59.00  -1.516
## first_speaker_typecautious                      56.68      16.62  59.00   3.410
## conditioncautious:first_speaker_typecautious   -13.92      15.61  60.00  -0.891
##                                              Pr(>|t|)    
## (Intercept)                                   0.38744    
## conditioncautious                             0.00058 ***
## test_orderparallel                            0.13477    
## first_speaker_typecautious                    0.00118 ** 
## conditioncautious:first_speaker_typecautious  0.37634    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) cndtnc tst_rd frst__
## conditincts  0.000                     
## tst_rdrprll -0.032  0.000              
## frst_spkr_t  0.000  0.000  0.000       
## cndtncts:__  0.000  0.000  0.000  0.000
## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: likelihood_ratio ~ condition + test_order + first_speaker_type +  
##     prior_likelihood_ratio + first_speaker_type * condition +  
##     (1 | workerid)
##    Data: d.post_test
## 
## REML criterion at convergence: 1587.1
## 
## Scaled residuals: 
##      Min       1Q   Median       3Q      Max 
## -2.39818 -0.66926 -0.04121  0.51524  3.06625 
## 
## Random effects:
##  Groups   Name        Variance Std.Dev.
##  workerid (Intercept)     0      0.0   
##  Residual             29267    171.1   
## Number of obs: 124, groups:  workerid, 62
## 
## Fixed effects:
##                                              Estimate Std. Error       df
## (Intercept)                                   16.4539    17.6008 118.0000
## conditioncautious                            -56.7419    15.3630 118.0000
## test_orderparallel                           -20.5737    15.4247 118.0000
## first_speaker_typecautious                    49.0672    15.5072 118.0000
## prior_likelihood_ratio                         0.4202     0.1165 118.0000
## conditioncautious:first_speaker_typecautious -13.9165    15.3630 118.0000
##                                              t value Pr(>|t|)    
## (Intercept)                                    0.935 0.351779    
## conditioncautious                             -3.693 0.000337 ***
## test_orderparallel                            -1.334 0.184835    
## first_speaker_typecautious                     3.164 0.001979 ** 
## prior_likelihood_ratio                         3.607 0.000455 ***
## conditioncautious:first_speaker_typecautious  -0.906 0.366866    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) cndtnc tst_rd frst__ prr_l_
## conditincts  0.000                            
## tst_rdrprll  0.013  0.000                     
## frst_spkr_t -0.066  0.000 -0.011              
## prr_lklhd_r  0.487  0.000  0.083 -0.136       
## cndtncts:__  0.000  0.000  0.000  0.000  0.000
## optimizer (nloptwrap) convergence code: 0 (OK)
## boundary (singular) fit: see help('isSingular')
## Data: d.post_test
## Models:
## model1: likelihood_ratio ~ condition + test_order + first_speaker_type + first_speaker_type * condition + (1 | workerid)
## model2: likelihood_ratio ~ condition + test_order + first_speaker_type + prior_likelihood_ratio + first_speaker_type * condition + (1 | workerid)
##        npar    AIC    BIC  logLik deviance  Chisq Df Pr(>Chisq)    
## model1    7 1647.8 1667.5 -816.89   1633.8                         
## model2    8 1637.0 1659.5 -810.49   1621.0 12.792  1  0.0003481 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

List of adapters:

workerid first_speaker_type test_order noticed_manipulation cautious_count confident_count aligned_count first_adaptation_speaker_count
1352 cautious reverse 1 1 1 2 1
1353 confident parallel 0 1 1 2 1
1354 cautious parallel 0 1 1 2 1
1355 confident reverse 0 1 1 2 1
1359 confident reverse 0 1 1 2 1
1365 cautious reverse 0 1 1 2 1
1368 confident parallel 1 1 1 2 1
1370 confident parallel 1 1 1 2 1
1373 cautious reverse 1 1 1 2 1
1375 confident reverse 1 1 1 2 1
1385 cautious reverse 1 1 1 2 1
1387 cautious reverse 1 1 1 2 1
1391 cautious reverse 1 1 1 2 1
1395 cautious parallel 1 1 1 2 1
1397 cautious reverse 1 1 1 2 1
1407 confident reverse 0 1 1 2 1
1417 cautious reverse 1 1 1 2 1
1423 cautious parallel 1 1 1 2 1
1428 cautious parallel 1 1 1 2 1
1432 confident reverse 0 1 1 2 1

List of reverse adapters:

workerid first_speaker_type test_order noticed_manipulation cautious_count confident_count aligned_count first_adaptation_speaker_count
1404 confident parallel 1 1 1 0 1
1406 cautious parallel 0 1 1 0 1
1419 confident reverse 0 1 1 0 1